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ABSTRACT 


The integration of self-driving vehicles introduces a unique and unprecedented 
human-machine interface that brings promise and peril. Several socially constructed 
theories try to explain this human-intelligent machine interface and predict how the future 
will look. This thesis offers a counter-narrative called Brown’s Point that suggests an 
alternative way of thinking about this relationship. The first Autopilot fatality offers a 
window into the human considerations needing attention as these intelligent machines, 
such as self-driving vehicles, combine with humans. How can the human-machine 
interface be optimized to ensure it offers the most benefit and safety for humanity? This 
thesis investigated the causal variables that led to the first Autopilot fatality by using 
Joshua Brown’s interface with the technology before and during the accident. I combined 
the findings from the accident investigation with various heuristics regarding the 
human-machine interface, theories from cognitive psychology, and sociological 
constructs to determine how Brown came to trust a machine he knew was fallible. 
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EXECUTIVE SUMMARY 


On May 7, 2016, at approximately 4:36 PM, my dear friend, Joshua D. Brown, 
was killed in his Tesla Model-S, while using an automatic driver-assistance technology. 
A semi truck made a left turn in front of his vehicle. The car impacted the elevated trailer 
at windshield height. The force of the impact sliced the car in half, instantly killing Josh. 

According to Tesla, a self-driving feature known as Autopilot was engaged at the 
time of the accident. i Tesla reports indicate that neither the Autopilot nor the driver had 
responded to the serni.^ According to the National Transportation Safety Board (NTSB), 
“While evidence revealed the Tesla driver was not attentive to the driving task, 
investigators could not determine from available evidence the reason for his inattention.”^ 
On the surface, it is easy to take a binary view of fault—either Brown failed to pay 
attention or given the car was supposed to drive itself, the Autopilot failed. However, it is 
neither one nor the other, but rather the combination of both, that makes this case so 
interesting. 

Exploring the reason for Brown’s lack of attention is a very important step to help 
understand potential consequences, but more importantly, understanding the dynamics 
and consequences of combining humans and self-driving vehicles allows a better 
understanding of how to keep this new relationship as safe as possible. Autopilot was in 
its first generation of the complex technology and Brown was an early adopter. In 
addition, no dedicated or mandatory training was required for the early adopter to 
understand the technologies’ capabilities and limitations; therefore, new and unexpected 
consequences could arise out of this unprecedented human-tech interface. 


^ The Tesla Team, comment, June 30, 2016, on Tesla.com, “A Tragic Loss,” Tesla Blog, June 30, 
2016, https://www.tesla.com/blog/ tragic-loss. 

2 Ibid. 

3 “Driver Errors, Overreliance on Automation, Lack of Safeguards, Led to Fatal Tesla Crash,” 
National Transportation Safety Board, September 12, 2017, https://www.ntsb.gov/news/press-releases/ 
Page S/PR20170912.aspx. 
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Three things are in play with the introduction of self-driving vehicles: 


1. Advancing technology (intelligent technology) 

2. A user trying to adapt to the new technology 

3. The interface between humans and self-driving vehicles that is yet to be 
fully explored 

This interface has, to date, not been the focus of a discreet entity. However, as 
technology becomes increasingly sophisticated and complex, the interaction between 
technologies and its intended user need to be understood fully and developed in its own 
right. This component is vital to implementing complex new technologies safely and is 
the focus of this thesis. 

The ongoing debate regarding ethics, which proposes to govern the spectrum of 
intelligent agents (e.g., self-driving vehicles), is critical to developing future realities 
safely for this emerging technology but just ethically programming machines will offer 
limited solutions. Humans will need to be programmed differently to accommodate the 
techno-centric future. 

This thesis sets out to understand better where the gaps are within the human- 
machine interface by using Brown’s case study to discover some of the answers. Having 
a better understanding of trust and automation bias, the heuristics and social commentary 
provides an opportunity to influence these variables moving into the future. 

Given that heuristics are socially constructed, tensions certainly have arisen 
among the fundamental considerations of ethics, intelligent technology, and 
neuroscience. Heuristics often lead to cognitive biases but offer a window into 
understanding decisions and problem solving.^ If carefully considered, it is possible to 
rehabilitate humans as autonomous self-actualizing beings who have a say in how it goes 
and what it means upon moving into a future full of artificial intelligence realities. At its 
conclusion, this thesis infers some human and technological considerations are needed as 

^ Wikipedia, s.v. “heuristics in judgment and decision-making,” last modified June 22, 2017, https:// 
en.wikipedia.org/wiki/Heuristics_injudgment_and_decision-making. 
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society moves into future realities and presents a counterfactual account of the events 
leading to Brown’s death. Altering the variables in a positive way shows the value in this 
approach to getting the human-machine integration right, and ultimately, reaching a new 
social commentary offered by the author called Brown’s Point. It serves as a tipping point 
and positive beacon for humanity that puts the human at the center of the human- 
technology relationship by combining the best of both technology and humanity.^ This 
synergistic relationship is achieved when technology augments humans and fills the gaps 
created by human biases that may take humanity to the next level of thinking. Brown’s 
Point is where the human-technological interface reaches the beginning of its potential. 

By taking a very techno-centric approach to improve efficiency and ease of work 
over the last century, society is experiencing the fruits of its labor. Machines are meant to 
help humans and improve the quality of life. Unfortunately, focusing so much on 
technology, the human in the equation appears to hold less importance. Some may argue 
the genie is out of the bottle and it is no longer possible to put it back.^ Technology is on 
a path that cannot be stopped. I disagree. It is possible to rehabilitate humanity but not 
alone. Partnering with technology and learning new innovative ways to help humans 
adapt quicker can help decrease the transition (i.e., hybrid) phase that will ultimately help 
society reach Brown’s Point and see the intention of advancing technology realized. 

The idea of Brown’s Point is applicable for any new advancing technology meant 
to interface with humans. Ensuring the variables and relationships between each are 
understood will help to design safe and helpful technology. It is possible with the speed 
of technology development that a reliable self-driving vehicle may come to fruition in the 
near future. Ensuring humans are as prepared as possible for this transition will allow 
self-driving vehicles to reach Brown’s Point, which is fantastic news for the safety of this 
nation’s roads and promises to save thousands of lives each year. 


5 Everett M. Rogers, Diffusion of Innovations, 5th ed. (Riverside: Free Press, 2003). 

6 Nick Bostrom, Superintelligence: Paths, Dangers, Strategies (Oxford, United Kingdom: Oxford 
University Press, 2014). 
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I. AUTOPILOT, THE FIRST FATALITY, AND ETHICS 
REGARDING INTELLIGENT MACHINES 

On May 7, 2016, at approximately 4:36 PM, my dear friend, Joshua D. Brown, 
was killed in his Tesla Model-S, while using an automatic driver-assistance technology. 
A semi truck made a left turn in front of his vehicle. The car impacted the elevated trailer 
at windshield height. The force of the impact sliced the car in half, instantly killing Josh. 

According to Tesla, a self-driving feature known as Autopilot was engaged at the 
time of the accident. i Tesla reports indicate that neither the Autopilot nor the driver had 
responded to the semi.2 According to the National Transportation Safety Board (NTSB), 
“While evidence revealed the Tesla driver was not attentive to the driving task, 
investigators could not determine from available evidence the reason for his inattention.”^ 
On the surface, it is easy to take a binary view of fault—either Brown failed to pay 
attention or given the car was supposed to drive itself, the Autopilot failed. However, it is 
neither one nor the other but rather the combination of both that makes this case so 
interesting. 

Exploring the reason for Brown’s lack of attention is a very important step to help 
understand potential consequences, but more importantly, understanding the dynamics 
and consequences of combining humans and self-driving vehicles allows a better 
understanding of how to keep this new relationship as safe as possible. Autopilot was in 
its first generation of the complex technology and Brown was an early adopter. In 
addition, no dedicated or mandatory training was required for the early adopter to 
understand the technology’s capabilities and limitations; therefore, new and unexpected 
consequences will arise out of this unprecedented human-tech interface. 


^ The Tesla Team, comment, June 30, 2016, on Tesla.com, “A Tragic Loss,” Tesla Blog, June 30, 
2016, https://www.tesla.com/blog/ tragic-loss. 

2 Ibid. 

3 “Driver Errors, Overreliance on Automation, Lack of Safeguards, Led to Fatal Tesla Crash,” 
National Transportation Safety Board, September 12, 2017, https://www.ntsb.gov/news/press-releases/ 
Pages/PR20170912.aspx. 
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The loss of my dear friend is undeniably tragic; however, I hope to learn from 
Josh’s death and offer ways of improving how humans accommodate the new techno¬ 
centric realities of the future. Determining the human considerations as society moves 
into these futures gives meaning to Brown’s death. Considering self-driving technology 
was developed to help improve safety, taking the time to analyze and discover the root 
causes of Brown’s lack of attention could help save the lives of others as self-driving 
technology integrates into society. Three things are in play: 

1. Advancing technology (intelligent technology) 

2. A user trying to adapt to the new technology, 

3. The interface between humans and self-driving vehicles that is yet to be 
fully explored 

This interface has to date not been the focus of a discreet entity. However, as 
technology becomes increasingly sophisticated and complex, the interaction between 
technologies and its intended user needs to be understood fully and developed in its own 
right. This component is vital to implementing complex new technologies safely and is 
the focus of this thesis. 

How humans interact with these highly specialized vehicles drastically changes 
what is known about the role of drivers. When combining humans and intelligent 
technology (machines), such as self-driving vehicles, how people think about this 
relationship requires contemplation and perhaps a paradigm shift. This warning does not 
suggest that the human-machine interface has only negative consequences. To the 
contrary, self-driving technology in vehicles is just the beginning of a new era of 
intelligent machine-human interfaces that brings incredible safety opportunities and 
exponential potential. What can be learned from the first associated fatality may have a 
lasting positive effect on intelligent technology moving forward.^ 
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Driver Errors, Overreliance on Automation, Lack of Safeguards, Led to Latal Tesla Crash.” 

2 



A. PROBLEM STATEMENT 

How can the human-machine interface be optimized to ensure it offers the most 
benefit and safety for humanity? According to three-time Pulitzer Prize winner Thomas 
L. Friedman, a gap exists between technology’s progress and the human’s ability to adopt 
and adapt to new technology.^ With technology touching nearly all facets of life in 
today’s world, machines using artificial intelligence (AI) are quickly becoming a reality. 
Humans have a vested interest in getting the future correct.^ If humans fail, they may 
write themselves out of history completely.^ Stephen Hawking believes the creation of AI 
“may pose the greatest existential threat to humanity.’’^ In his warning. Hawking 
imagines a world where silicon replaces carbon, computers take over, and humans 
become obsolete; a terrifying thought. Though the argument sounds very much a part of 
the science fiction realm because of the power and ability of intelligent machines, they 
have the ability to learn faster and store greater amounts of information than humans.^ 
Intelligent machines may far exceed the abilities of humans. Therefore, this narrative 
receives a great deal of attention. 

Unfortunately, the rate of change with technology creates challenges. Take for 
instance, a cell phone. As soon as humans adapt to the most recent iteration of a 
technology, for instance, the iPhone 7, a new and improved version is released—the 
iPhone 8, 10, ad infinitum. Except for a small number of early adopters, humans are 
generally behind the curve and always seem to be catching up to the latest technology. 
The fast-moving iteration of technology applies to self-driving vehicles as well. Humans 


^ Thomas L. Friedman, Thank You for Being Late: An Optimists Guide to Thriving in the Age of 
Accelerations (New York: Farrar, Straus, Giroux, 2016). 

^ Stephen Hawking, “Stephen Hawking: ‘Transcendence Looks at the Implications of Artificial 
Intelligence—but Are We Taking AI Seriously Enough?’” The Independent, May 1, 2014, http://www. 
independent.co.uk/news/science/stephen-hawking-transcendence-looks-at-the-implications-of-artificial- 
intelligence-but-are-we-taking-9313474.html. 

7 Ibid. 

8 Ibid. 

^ Lance Whitney, “Are Computers Already Smarter Than Humans?” Time, September 29, 2017, http:// 
time.com/4960778/computers-smarter-than-humans/. 
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are falling behind while intelligent machines continue to learn and get smarter. This 
situation may result in unintended consequences—even death. 

Another piece of the problem is the desire of innovators to be the first to build 
successful self-driving vehicles. Although winning the sprint to implementation may 
seem appealing, it is important for developers to slow down and consider the vast amount 
of variables as humans and machines begin to integrate. Due to the amount of attention, 
the promise of improved safety, and the integration of self-driving vehicles in the very 
near future, applying academic rigor to help understand human cognition and 
neuroscience allows for a much deeper dialogue about the way in which humans come to 
view their role and the decision-making process with intelligent machines (AI). 

While removing the human from behind the wheel may solve one problem, it also 
creates many new challenges. The majority of the academic papers written on self¬ 
driving vehicles and technology adoption focus on the timeline for development. In 
addition, ethical challenges, liability concerns, cyber security issues predominate, as does 
an emphasis on potential benefits, for instance, less impact on the environment, safety 
improvements, and quality of life for certain populations, Due to limited data and only 
anecdotal evidence, thus far, future contributions, speed of integration, and identification 
of human considerations is not well explored. This thesis argues a greater focus should be 
placed on the human variables, not just the technology. Human considerations are 
necessary for optimizing the outcomes of combining humans with intelligent machines, 
and in this thesis, self-driving vehicles are an excellent starting point. 

The first Autopilot fatality serves as a unique case study and starting point to 
identify contributing human factors leading to Brown’s death. By understanding the 
factors, social commentary, and perhaps gaps of the self-driving vehicle, an opportunity 
exists to address them, and potentially, rectify future problems. 


Melinda Fiorina Lohmann, “Liability Issues Concerning Self-driving Vehicles,” European Journal 
of Risk Regulation 7, no. 02 (2016): 335, doi: 10.1017/sl867299x00005754; Jason Millar, “An Ethics 
Evaluation Tool for Automating Ethical Decision-Making in Robots and Self-Driving Cars,” Applied 
Artificial Intelligence 30, no. 8 (2016): 787, doi: 10.1080/08839514.2016.1229919. 
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B. RESEARCH QUESTION 

How can the human-machine interface be optimized to ensure it offers the most 
benefit and safety for humanity? 

C. LITERATURE REVIEW 

To educate in mind and not in morals is to educate a menace to society. 

~ Theodore Roosevelt^^ 

This section analyzes relevant literature in the field of ethics in robotics, or AI 
agents, as technology advances and machines learn autonomously. Programming 
machines is an important piece to AI futures. The term intelligent agents allows for a 
broader spectrum of intelligent machines, not simply the quintessential form of humanoid 
robots made popular by science fiction. Intelligent agents come in many forms, from 
large industrial machines to self-driving vehicles to such simple devices as the Amazon 
Echo. Given their ability to learn and act autonomously, it is important to understand how 
and why machines choose solutions to problems and how humans interact with them. The 
ongoing debate regarding ethics, which proposes to govern the spectrum of intelligent 
agents, is critical to developing future realities safely for this emerging technology. This 
literature review discusses the historical and modern theories centered on ethical 
considerations for intelligent agents. 

What is known about robotics, until recently, developed mostly out of science 
fiction novels and movies. The idea of robots, which assist humans and provide support 
in their day-to-day activities, is generations old. 12 As technology advances at an 
exponential rate, society is beginning to reap the benefits of intelligent machines. 


Theodore Roosevelt, “Theodore Roosevelt Quotes,” BrainyQuote, accessed September 28, 2017, 
https://www.brainyquote.eom/quotes/quotes/t/theodorerol47876.html. 

Nayef Al-Rodhan, “The Many Ethical Implications of Emerging Technologies,” Scientific 
American, 4, March 13, 2015, https://www.scientificamerican.com/article/the-many-ethical-implications- 
of-emerging-technologies/; Nick Bostrom, Superintelligence: Paths, Dangers, Strategies (Oxford, United 
Kingdom: Oxford University Press, 2014). 
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However, many scientists warn of rogue intelligent machines, which may harm 
humans if scientists fail to provide safeguards in their development^^ In his 2014 book. 
Superintelligence: Paths, Dangers, Strategies, Nick Bostrom, director of the Future of 
Humanity Institute at the University of Oxford, warns of the need for safeguards. What it 
comes down to, he states, is power. The fate of humanity may rest in what a 
superintelligent agent wants, not what humans want.^^ Bostrom predicts the invention of 
machine intelligence will be the last invention humans will ever need to make because 
machines will be inventing better than people, and they will do so on digital time 
scales. Bostrom emphasizes that if humans “create a really powerful optimizing 
intelligent agent for X reason it is critical that we include everything humans care about 
in defining X.”i6 He compares a super-intelligent agent to a genie in a bottle and believes 
“we must anticipate humans will not succeed in confining this genie to its bottle forever; 
therefore, the intelligent agent must be fundamentally on our side and share human 
values.”!^ His solution includes humans solving this control problem in advance, so if 
and when it is needed, it is readily available.!^ 

Stephen Hawking also suggests the new progression of research into intelligent 
agents has not been taken seriously enough. The creation of AI may be the biggest 
event in human history; however, it may also pose the biggest existential threat to 
humanity .20 He suggests, “All of us should ask ourselves what we can do now to improve 
the chances of reaping the benefits and avoiding the risks.”2i Striving for Hawking’s 


^5 Bostrom, Superintelligence: Paths, Dangers, Strategies. 

Nick Bostrom, “What Happens When Our Computers Get Smarter Than We Are?” TED2015, 
video, 16:31, March 2015, https://www.ted.com/talks/nick_bostrom_what_happens_when_our_computers_ 
get_smarter_than_we_are#t-451360. 

15 Ibid., 07:55. 

16 Ibid., 09:24. 

12 Ibid., 13:25. 

18 Ibid., 15:46. 

19 Hawking, “Stephen Hawking: ‘Transcendence Looks at the Implications of Artificial Intelligence— 
but Are We Taking AI Seriously Enough?’” 

20 Ibid. 

21 Ibid. 
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suggestion will require dedicated researchers to commit to a better understanding on how 
to accomplish this task. 

The father of the robotic paradigm and a biochemist by trade, Isaac Asimov wrote 
his influential book named 1 Robot in 1950.22 Asimov assumed programming robots with 
moral laws could control their behavior and decisions.23 Therefore, humans should 
ensure robots and intelligent agents follow legislation and rules based on human ethics.24 
Although it can be argued that Asimov’s work is simply entertainment, in this case, it 
appears he used storytelling as social commentary. He warns that technology may 
encroach on the human domain.25 Asimov’s writing provides a narrative that has 
significantly influenced society’s hopes and fears of robotics and intelligent agents. To 
protect humans from the encroaching technology, Asimov asserts these electronic beings 
are in need of ethical programming.26 

The primary concern he raises is that without programmed ethics, the autonomous 
decisions of robots may have devastating effects on humanity .22 Theoretically, he 
believes that if machines are not given boundaries, they may rise and pose an existential 
threat to all humanity. To counter this menace, Asimov proposed a synthetic values 
system for robots by creating three laws to govern their behavior: 

1. A robot may not injure a human being, or through inaction, allow a human 
being to come to harm. 

2. A robot must obey the orders given it by human beings, except where such 
order would conflict with the first law. 


22 Isaac Asimov, I Robot (Garden City, NY: Doubleday, 1950). 

23 Ibid. 

24 Hawking, “Stephen Hawking: ‘Transcendence Looks at the Implications of Artificial Intelligence— 
but Are We Taking AI Seriously Enough?’”; Bostrom, Superintelligence: Paths, Dangers, Strategies', 
Asimov, I Robot. 

25 Asimov, I Robot. 

26 Ibid. 

22 Ibid. 
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3. 


A robot must protect its own existence as long as such protection does not 
conflict with the first or second law 


Asimov’s legacy has influenced engineering and robotic communities for 
decades. David Camarillo, with the Department of Electrical Engineering at Stanford 
University, writes about Asimov’s legacy and how it influenced the introduction of 
robotics in surgical rooms in the 1980s.^9 Camarillo claims Asimov’s three laws “remain 
a reasonable ethical framework for the development of robots as applied to surgical 

care.”30 

Nayef Al-Rodhan, author, neuroscientist, and honorary fellow at the University of 
Oxford in England, supports Asimov’s concerns of technology outrunning regulations 
and justifies why ethics are so important in this emerging technology.31 He argues it is 
imperative society sees both sides of technology.32 The positive and negative, or dual 
use, aspects of emerging technology raise questions of standardization, traceability, and 
copyright. 33 Al-Rodham suggests that although modern technology development 
demonstrates human excellence, many ethical red flags have emerged. 34 Al-Rodham 
admits the scientific community does not fully understand the consequences, and good 
governance plays a role in the dual use of technology.35 If negative dual uses of 
technology are preventable through strong regulation, it should be seriously considered, 
as long as regulation improves safety but does not hinder innovation. 


2^ Asimov, I Robot. 

29 David B. Camarillo, Thomas M. Krummel, and J. Kenneth Salisbury, “Robotic Technology in 
Surgery: Past, Present, and Future,” American Journal of Surgery 188, no. 4 (2004): 2S, doi: 10.1016/ 
j.amjsurg.2004.08.025. 

30 Ibid. 

31 Nathan Bomey and Thomas Zambito, “Regulators Scramble to Stay Ahead of Self-driving Cars,” 
USA Today, June 25, 2017, https://www.usatoday.eom/story/money/cars/2017/06/25/regulators-scramble- 
stay-ahead-self-driving-cars/100963150/. 

32 Al-Rodhan, “The Many Ethical Implications of Emerging Technologies,” 4. 

33 Ibid., 5. 

34 Ibid. 

35 Ibid. 
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On the one hand, many researchers boast of the positive benefits robots will bring 
for humanity. For example, self-driving vehicles will improve the quality of life for the 
disabled and elderly population. On the other hand, it is important to understand how 
robots may rise to power, as well as outsmart and threaten the human domain. 36 Also, 
actors who wish to use intelligence agents for nefarious intentions raise a valid concern. 
Crime, terrorism, cyber security are potential threats and are an example of this dual-use 
conundrum.37 Self-driving vehicles are designed to improve safety; however, if a terrorist 
packs a car full of explosives and programs it to drive to location X where it is detonated, 
obviously the vehicle is not being used for its intended purpose. Al-Rodham does not 
offer a framework to address his concerns. Rather, he claims a need exists for deeper 
exploration of ethics and dual uses of technology to understand fully the consequences 
possibly faced in the future. 

Building on Asimov’s and Rodhan’s ideas, Headleand and Teahand also address 
the dual use of technology and support the idea that as AI requires less human 
supervision and grows more autonomous, developing ethical smart systems is critical to 
averting catastrophic consequences of rogue intelligent agents.38 Similar to Asimov, 
Headleand and Teahand believe as machines become smarter, they may impact the rights 
of humans.39 They consider ethical decisions from egotistical and altruistic perspectives. 
In their experiments using Braitenberg’s vehicles—known as a major cornerstone of 
robotics research—they observe intelligent agents that use only one of the two 
perspectives exhibit self-preservation problems. As a result, they conclude that striking a 
balance between the egotistical and the altruistic allows the strengths of one to help 
alleviate the weakness of the other.^o With this balance in mind, Headleand and Teahand 


36 Al-Rodhan, “The Many Ethical Implications of Emerging Technologies,” 5. 

3^ Department of Homeland Security, The 2014 Quadrennial Homeland Security Review (Washington, 
DC; Department of Homeland Security, 2014), 6, https://www.dhs.gov/sites/default/files/publications/qhsr/ 
2014-qhsr.pdf. 

38 Christopher J. Headleand and William Teahand, “Towards Ethical Robots: Revisiting Braitenberg’s 
Vehicles,” SAI Computing Conference 2016, London, UK, July 13-15, 2016, 469. 

39 Ibid. 

40 Ibid. 
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propose replacing Asimov’s three laws with two laws of their own, which focus more on 
self-preservation of the intelligent agent: 

1. A vehicle [intelligent agent] must act to preserve its existence. 

2. A vehicle [intelligent agent] must not prevent another from self- 
preservation, except in situations where any sacrifice would conflict with 
the first rule.^i 

Muehlhauser and Helm reject the idea that intelligent agents should be 
programmed with ethical rule-based constraints. They propose that super-optimized 
computers (intelligent agents) will simply go around the rules.^2 Taking a more human¬ 
centric approach to ethics in emerging technology, Muehlhauser and Helm approach the 
challenge of machine ethics in their work by suggesting that machines should be 
programmed to learn and abide by the motivations or goals of humans. However, the 
problem they foresee with this model is that humans are not sure what they want, thus 
making the programming of intelligent agents challenging.43 

Meuhlhauser and Helm point out, “Neuroscientific and behavioral evidence 
suggests moral thinking is largely an emotional process and may in most cases amount to 
little more than a post hoc rationalization of our emotional reactions to situations. 
Considering the varying and inconsistent choices humans make, Meuhlhauser and Helm 
ask the question, “if these choices are a result of emotions or competing value systems, 
are choices more dependent on framing rather than the content of the options ?”45 If 
science can select one value system as the preferred system, programming intelligent 
agents is easy. Nevertheless, humans’ neurobiology must be understood to accomplish 


Headleand and Teahand, “Towards Ethical Robots; Revisiting Braitenberg’s Vehicles,” 469. 

42 Amnon H. Eden and Eric Steinhart, Singularity Hypotheses: A Scientific and Philosophical 
Assessment (Heidelberg: Springer, 2012), 102. 

43 Ibid. 

44 Ibid., 111. 

45 Ibid., 115. 
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this task.46 They leave their work open for others to explore how humans determine their 
frame of reference. 


While most past research focuses on a robotic-centric approach to programming 
intelligent agents, Nasraoui and Shafto point to the importance of the interaction of 
humans with machines and how this interaction may affect machine leaming.47 Their 
research has value, as it acknowledges the input received from the human-machine 
interaction determines the output (choices) of the machine. Nasraoui and Shafto point out 
that ethical programming of intelligent machines is flawed because algorithms are static 
yet interactions with humans are dynamic.48 If an intelligent agent learns through 
experience and human-machine interactions, what it learns may be biased. These biased 
inputs may lead to negative consequences.49 Nasraoui and Shafto’s conclusions suggest 
ongoing research is needed to consider a framework from which to study the human- 
algorithm interaction.60 

Murphy and Woods also challenge Asimov’s laws, asking whether the framework 
for human-machine interaction is viable or simply a cultural narrative with little empirical 
evidence.6i They review the possible shortcomings and dangers of each law due to the 
complexities and dynamics of relationships between people and intelligent agents.62 
Murphy and Woods offer a parallel set of laws to Asimov’s original three. The difference 
is that their framework focuses primarily on the human ethics behind intelligent agent 
technology. They argue a robot’s ethics are not nearly as important as the ethics of those 


46 Eden and Steinhart, Singularity Hypotheses: A Scientific and Philosophical Assessment, 115. 

42 Olga Nasraoui and Patrick Shafto, “Human-Algorithm Interaction Biases in the Big Data Cycle: 
Markov Chain Iterated Learning Framework,” Computing Research Repository (CoRR) Abs/1608.07895, 
September 2, 2016, https://www.researchgate.net/publication/307302992_Human-Algorithm_lnteraction_ 
Biases_in_the_Big_Data_Cycle_A_Markov_Chain_lterated_Learning_Framework. 

48 Ibid. 

49 Ibid. 

50 Ibid. 

51 Robin Murphy and David D. Woods, “Beyond Asimov: The Three Laws of Responsible Robotics,” 
IEEE Intelligent Systems 24, no. 4 (2009): 14, doi: 10.1109/mis.2009.69. 

52 Ibid., 19. 
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developing, testing, and deploying the emerging technology. ^3 Murphy and Woods’s 
parallel laws are as follows: 

1. A human may not deploy a robot without the human-robot work system 
meeting the highest legal and professional standards of safety and ethics. 

2. A robot must respond to humans as appropriate for their roles. 

3. A robot must be endowed with sufficient situated autonomy to protect its 
own existence, as long as such protection provides a smooth transfer of 
control to other agents consistent with the first and second laws.34 

As Murphy and Woods suggest, it is more important to focus on the human in the 
dynamic human-machine relationship.^^ Understanding the human elements when 
integrating humans and intelligent machines takes much more effort than simply 
programming an intelligent agent. Can humans be programmed, should they be, and if so, 
how? Muehlhauser’s suggestion that morals are possibly an emotional reaction to a 
situation may mean the ability to change the framing of the human-machine 
relationship.36 Just ethically programming machines will offer limited solutions. Humans 
will need to be programmed differently to accommodate the techno-centric future. 

D. RESEARCH DESIGN 

The goal of this study is to identify human considerations needed for the future of 
AI realities, particularly regarding the self-driving vehicle revolution. Using the first 
Autopilot fatality as a case study, I apply theories from expert publications and scholarly 
papers to help analyze ethics, technology adoption theories, cognitive biases, and 
heuristics to find solutions associated with the human-machine relationship. 

What makes this case study particularly useful is that Brown documented his 
growing comfort and trust in Autopilot’s ability to keep him safe. Analyzing the 

33 Murphy and Woods, “Beyond Asimov: The Three Laws of Responsible Robotics,” 19. 

54 Ibid., 17-18. 

55 Ibid. 

5^ Eden and Steinhart, Singularity Hypotheses: A Scientific and Philosophical Assessment, 102. 
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chronological documentation of comments and videos Brown published on his personal 
social media accounts can assist readers in understanding the development of his 
relationship and heuristics about Autopilot. Understanding the causal mechanisms can 
help explain contributing factors leading to his death and possible human and 
technological consideration needs for future techno-centric realities.By focusing on 
causal-process observations, I can get inside the “black box” of decision making and 
explore perceptions and expectations, both to explain his individual historical experiences 
and suggest more generalizable hypotheses.^8 

Using Teller’s ideas regarding technology’s rate of change and humanity’s ability 
to adapt to technology, an alternative theory is offered called “Brown’s Point.”59 Brown’s 
Point illustrates a point in time when humans and technology partner to create 
exponentially positive results. Cognitive biases and trust development theories are used to 
analyze the months leading up to the fatal crash and Brown’s interaction with the 
autopilot and auto-steer features in his Tesla to demonstrate the socially constructed 
heuristics he developed using the self-driving technology. The most important questions 
are why Brown trusted a vehicle he knew had limitations and how can the why be 
prevented in the future? I propose the why can be explained through social constructs, 
heuristic development, and biases, such as automation and confirmation bias bringing 
awareness to the need to take a human-centric approach in the human-machine 
interface. 

Given that heuristics are socially constructed, tension certainly has arisen among 
the fundamental considerations of ethics, intelligent technology, and neuroscience. 
Heuristics often lead to cognitive biases but offer a window into understanding decisions 


Eden and Steinhart, Singularity Hypotheses: A Scientific and Philosophical Assessment, 6. 

68 Ibid. 

69 Friedman, Thank You for Being Late: An Optimists Guide to Thriving in the Age of Accelerations. 

60 Richard Breton and Eloi Bosse, “The Cognitive Costs and Benefits of Automation” (lecture, RTO 
HEM Symposium, Warsaw, Poland, July 14, 2016). 
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and problem solving.If carefully considered, it is possible to rehabilitate humans as 
autonomous self-actualizing beings who have a say in how it goes and what it means 
when moving into a future full of AI realities. At its conclusion, this thesis infers some 
human and technological considerations needed as society moves into future realities and 
presents a counterfactual account of the events leading to Brown’s death. By altering the 
variables in a positive way, this study shows value in this approach to getting the human- 
machine integration right and ultimately reaching Brown’s Point. 

The road map for the remainder of this thesis is as follows. Chapter II explains 
who Joshua Brown was and the timeline of the accident. It also looks at the contributing 
factors that led to the first Autopilot fatality, as well as the human responsibility in 
developing the human-machine interface narrative. Chapter III lays out an explanation of 
the current state of technology and human adaptability, as well as an ominous prediction 
of humanity’s future as technology advances. Chapter IV offers an alternative narrative to 
the theories explored in Chapter III by defining the transition period as the hybrid phase 
and a way forward to a point in time when human’s combining with technology reali z es 
the exponential potential of this relationship. This point in time is called Brown’s Point. 
Finally, in Chapter V, Brown’s story is rewritten as if the contributing factors leading to 
his death are fixed and result in an alternative outcome. The deliverable in this thesis is 
hypotheses on how technology and humanity can help narrow the gap between 
technology’s rate of change and human’s ability to adapt. It ends with suggesting further 
areas of study are needed. 


“In Memoriam: Noted Researcher and Scholar Raja Parasuraman,” George Mason University, 
March 26, 2015, https://www2.gmu.edu/news/1443; Raja Parasuraman and Victor Riley, “Humans and 
Automation: Use, Misuse, Disuse, Abuse,” Human Factors: The Journal of the Human Factors and 
Ergonomics Society 39, no. 2 (June 1997): 249. 
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II. WHO IS REALLY IN CHARGE: THE ACCIDENT ANALYSIS 


A press release by the Brown family from September 11, 2017 after the 
completion of the accident investigation provided a summary of Josh’s career and life 
interest: 

Josh was a [Armed Services] veteran, an exceptional citizen, and a 
successful entrepreneur. Most importantly, he was a loving son, brother 
and uncle. Josh served 11 years in the United States Navy. He was a 
master Explosive Ordnance Disposal (EOD) technician and achieved the 
rank of Chief Petty Officer. He proudly served as a member of EOD 
Mobile Unit 3 out of San Diego, CA, and then the Navy’s elite Naval 
Special Warfare Development Group (NSWDG) out of Dam Neck, VA. 

Josh was deployed to multiple war zones as part of the special operations 
groups. He also served at the White House and overseas supporting Secret 
Service operations. 

Joshua loved technology and was a successful entrepreneur. He developed 
several database applications widely used by the Navy. In 2010, he started 
his own technology company, Nexu Innovations. The company primarily 
focused on developing and installing WIEI and surveillance systems, but 
also developed other technology driven applications.^2 

As the statement notes. Josh was no stranger to technology or the need for 
situation awareness. He made a concerted effort always to emphasize safety, especially 
while in his Tesla. In fact, according to a press release from his family on September 11, 
2017, they stated: 

Joshua loved his Tesla Model S. He studied and tested that car as a 
passion. When attending gatherings at the Tesla store, he would become 
the primary speaker answering questions about the technology and the 
car’s capabilities/limitations. In the videos Josh posted to YouTube about 
Tesla, he repeatedly emphasized safety, that the car was NOT 
autonomous, and that the driver had to pay attention. ^3 


62 Landskroner Grieco Merriman, LLC, “Statement from the Family of Joshua Brown,” news release, 
Cleveland, OH, 2017. 

63 Ibid. 
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While he was impressed with the ability of the autopilot to learn, acknowledging 
its limitation was a common theme for Brown’s Facebook posts Unfortunately, like all 
humans. Brown did not know what he did not know; an easy trap for anyone to fall prey 
to. Exploring the events leading up to his decision to trust Autopilot beyond its capability 
helps to understand human tendencies with the goal of becoming aware and offering 
recommendations to fix the problems. 

Josh owned his Tesla for just over 10 months when he was killed. In that short 
time period, he accumulated over 39,000 miles.65 Quite a feat considering that number is 
nearly triple the average American’s mileage per year. Obviously, Brown had a lot of 
time behind the wheel of his Tesla but less time with the Autopilot feature. The Autopilot 
feature was not rolled out to the Tesla fleet until November 2015. Thus, Brown had 
approximately six months to learn about and use the first iteration of the Autopilot 

technology. 66 

A. THE DETAILS OF THE ACCIDENT 

Although the vehicle and the driver were very capable in and of themselves, they 
both failed to recognize the danger, pointing fault to the human-machine interface. 

The weather was clear and dry on May 7, 2016.6^ At approximately 4:36 PM, 
Brown was traveling eastbound on U.S. 27 just west of Williston, Florida in his 2015 
Tesla Model S.68 Traveling in the opposite direction on the same highway was an 

64 Comment on “Milage on Tessy,” Joshua D. Brown Facebook page, December 8, 2015, https://www. 
facebook.com/joshua.brown. 16940/posts/10153676267625734. 

65 Joshua Brown, “Autopilot Saves Model S,” Joshua Brown YouTube comments, 0;41, April 05, 

2016, https://www.youtube.com/watch?v=9I5rraWJq6E. 

66 Ryan Bradley, “10 Breakthrough Technologies 2016: Tesla Autopilot,” MIT Technology Review, 
February 21, 2017, https://www.technologyreview.com/s/600772/10-breakthrough-technologies-2016- 
tesla-autopilot/. 

6^ Daphne Yuncker, Florida Highway Patrol Trajfic Homicide Field Note Packet-THI Case No. 716- 
39-007 (Williston, FL: Florida Highway Patrol, 2016), https://docs.wixstatic.com/ugd/63eaea_8284ad6 
f230e4e7daa0514ef78f75f63.pdf. 

68 National Highway Traffic Safety Administration, ODI Resume (Washington, DC: Department of 
Transportation, 2017), 1, https://www.bing.com/cr?IG=69C0F25AFFD74B89A0B9CDDA9CACD8DF& 
CID=348EF5F7E1516A0C1975EED8E0576BCC&rd= 1 &h=OqDXIJlhAqoYh-Yk28hyAuhZEtrqNBvK 
XGCI9_ZG7E&v=l&r=https%3a%2f%2fstatic.nhtsa.gov%2fodi%2finv%2f2016%2fINCLA-PE16007- 
7876.pdf&p=DevEx,5063.1. 
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elevated refrigerated semi truck carrying a load of blueberries. xhe truck driver exited 
the highway into the left turning lane and proceeded into the uncontrolled intersection in 
front of Brown’s Tesla.^® At 74 MPH, the car travelled underneath the rear third of the 
trailer shearing off the top of Brown’s car at windshield height.^i The Tesla continued 
traveling eastbound before veering off the highway into a field, hitting a utility pole, and 
coming to rest in a field.^2 Brown was instantly killed, and according to dispatch 
information, pronounced dead at approximately 4:51 PM.^3 

In the following 18 months, one of the most important accident investigations was 
underway. Self-driving technology promises to disrupt the entire transportation industry 
and potentially change how society moves people and goods. The findings of the 
investigation are likely to set precedence and modify the trajectory of the self-driving 
revolution for better, or for worse. Two separate governing bodies. The National 
Highway Traffic Safety Administration (NTHSA) and the NTSB have published findings 

from their investigations. 

The NTHSA Office of Defects Investigation (ODI) determined that: 

I) the Tesla was being operated in Autopilot mode at the time of the 
collision; 2) the Automatic Emergency Braking (AEB) system did not 
provide any warning or automated braking for the collision event; 3) the 
driver took no braking, steering or other actions to avoid the collision; and 
4) the last recorded driver action was increasing the cruise control set 
speed to 74 mph less than two minutes prior to impact.^5 


Yuncker, Florida Highway Patrol Trajfic Homicide Field Note Packet-THI Case No. 716-39-007. 

“Board Meeting Animation HWY16FH018,” YouTube video, 1:09, posted by NTSBgov, 

September 12, 2017, https://www.youtube.com/watch?v=YFykkqYGVfl&feature=youtu.be. 

Yuncker, Florida Highway Patrol Trajfic Homicide Field Note Packet-THI Case No. 716-39-007. 

72 Ibid. 

73 Interact, CAD Detail, Incident # 2016-00032596 (Williston, FL: Interact, 2016), https://docs.wix 
Static.eom/ugd/63eaea_45d414c8faf64033b94d44c3c23cd75b.pdf. 

74 National Transportation Safety Board, Collision between a Car Operating with Automated Vehicle 
Control Systems and a Tractor-Semitrailer Truck Near Williston, Florida May 7, 2016, NTSB/HAR-17/02 
(Washington, DC: National Transportation Safety Board, 2017), 41^2, https://www.bing.com/cr?IG=06 
4E1 DE6F2594080BCCDF6349ACBE808&CID=24EEE51EAC5B6C1 BOl 71EE30AD5D6DB1 &rd= 1 &h= 
jWD5JD23VPTeEjpq0w2d2o0N07LKmNm3HVT2tRnwTfo&v=l&r=https%3a%2f%2fwww.ntsb.gov%2f 
investigations%2fAccidentReports%2fReports%2fHAR1702.pdf&p=DevEx,5036.1. 

73 National Highway Traffic Safety Administration, ODI Resume, 1. 
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As shown is Figures 1-3, the NTSB determined the point where Brown’s car 
crested the hill until impact with the semi took approximately 10 seconds.Considering 
Brown did not take evasive maneuvers indicates he was unaware of the peril. It is known 
that he was distracted; a simple conclusion. What is not known and must be discovered is 
how and why Brown came to trust a machine he knew was fallible. Of interest, the NTSB 
acknowledged that the type of highway Brown was traveling on was understood not to be 
an optimal road on which to use the Autopilot feature; however, nothing was in place to 
prevent him from setting and using the system as he had done many times before. 



Figure 1. NTSB Collision Animation^^ The National Traffic Safety Board 
(NTSB) determined from the time Brown crested the hill until impact with 
the semi was about 10 seconds. 


“Board Meeting Animation HWY16FH018.” 

Board meeting to determine probably cause of the fatal. May 7, 2016, crash of a Tesla car near 
Williston, Florida. Deb Bruce, “Collision Between a Car Operating With Automated Vehicle Control 
Systems and a Tractor-Semitrailer Truck Near Williston, Florida May 7, 2016” (PowerPoint presentation, 
National Transportation Safety Board Meeting September 12, 2017), https://www.ntbs.gov/news/events/ 
Documents/2017-HWY 16FFI018-BMG-presentations.pdf 

Source: Ibid. 
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Figure 2. Tesla after Collision with Semi Truck^^ 



Figure 3. Semi Truck after Collision with Tesla^*^ 


Source: National Transportation Safety Board, Collision between a Car Operating with Automated 
Vehicle Control Systems and a Tractor-Semitrailer Truck Near Williston, Florida May 7, 2016, 6. 

Source: Ibid., 7. 
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As noted, Brown’s death was preceded by a series of cascading events that began 
with the naming of autopilot followed by a series of entirely human mistakes of thought 
and perception. This horrific accident is the result of frictions and noise at the level of the 
human-machine interface. Finding a solution to the friction and noise takes a greater 
understanding of why and how Brown developed such trust in a fallible machine. 

B. THE ACCIDENT ANALYSIS: TRUST, AUTOBIAS, AND WHO IS 

REALLY IN CHARGE 

Analyzing the accident using different measures and ideas helps bring 
understanding to the factors that contributed to Brown’s fatality. I propose that the words 
used to describe the innovative technology also bring unintended consequences, yet bring 
awareness to the importance of designing and naming innovations, with intent. Interfaces 
between humans and machines have dependent variables. Humans must learn to trust 
technology and technology must have trust to work as designed especially when it comes 
to the automation of tasks. Too much or too little trust affects decision making and—as in 
Brown’s fatality—leads to the challenge of human biases. 

I. Autopilot: What Is in a Name? 

The words chosen to define self-driving technology bring meaning and provide a 
frame from which to understand technology’s roles and capabilities better. For this 
reason, it is important to choose words that convey the correct frame and meaning. 
Arguably, “Autopilot,” a nickname, which stuck when first introduced by the Tesla team 
in October 2015, is misleading because it incorrectly implies the vehicle drives itself.^i 
Although quite advanced in its capability. Autopilot is far from having the ability to fully 
drive itself. Hence, California decided to create draft regulation to fix the problem with 
naming the feature. Autopilot. On October 4, 2016, “California DMV issued draft 
regulation for a cease from using the terms “Self Driving,” “Automated” or “Autopilot” 
in advertising unless the vehicle is a Level 4 autonomous vehicle. 

“15-Autopilot Fatality,” The Tesla Show—A Tesla Podcast (blog), Tesla Show, July 12, 2016, 
https://itunes.apple.com/us/podcast/the-tesla-show/idl 103101874?mt=2&i=372397989. 

82 “Pre-and Post-Fatality,” Amanda Lee, last edited September 24, 2017, https://alee649.wixsite.com/ 
joshuabrown. 
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Although this example is centered on Telsa, other developers of self-driving 
technology have run into similar challenges with the words and advertisement of their 
products. 

Mercedes faced criticism over a TV commercial for its 2017 E-class, arguing “the 
ad could mislead consumers by overstating the capability of automated driving systems 
available on the sedan.”^^ Mercedes executives call the driver-assist feature, which 
utilizes adaptive cruise control and automated steering, “Drivepilot,” similar to Tesla’s 
Autopilot. The German carmaker claims its technology is capable of keeping speeds of 
130 mph while in drive-pilot. The advertisement in question shows a driver with no 
hands on the wheel for the entire commercial. Mercedes appears to understand the 
conflict with its commercial because it added a subtle disclaimer at the bottom of the 
screen. It reads in part, “the vehicle cannot drive itself, but has automated driving 
features.Since the scrutiny, Mercedes has removed its advertisement, further 
supporting the claim that perceptions and expectations based on the advertisement of 
products do matter. 

Even though words, such as Autopilot or Drivepilot, are not intended to do harm, 
they can contribute to bias and heuristic development by framing how people think about 
technology’s capabilities. The power of suggestion is very important to understand and 
consider when framing and bringing meaning to new technology, which points to looking 
at both the design of the machine, its capabilities, and the human role in the human- 
machine interface. Trust is an important component of the human-machine interface but 
can also lead to a problematic relationship if the technology is not capable of what the 
human assumes the technology is capable of handling. 

2. Trust: Too Much of a Good Thing? 

The accident reconstruction indicates neither the vehicle nor the driver recognized 
the peril of a 53-foot trailer perpendicular to his path, and therefore, did not take evasive 

^3 Ryan Beene, “Mercedes Challenged over ‘Drive Pilot’ TV Ad,” Automotive News, July 27, 2016, 
http://www.autonews.eom/article/20160727/RETAIL03/160729855/mercedes-challenged-over-drive-pilot- 
tv-ad. 

84 Ibid. 
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actions to avoid the collision.n ig assumed Brown was not as active a participant 
in supervising the self-driving vehicle as necessary. It stands to reason he did not 
recognize any danger because he was trusting Autopilot to provide for his safety. 

Through a series of causes and effects, Brown learned to overtrust Autopilot. 
Even though trust is one of the goals of self-driving vehicles, when the trust pendulum 
swings too far in either direction, the risk of negative consequences increases.Most 
individuals who interact with new technology learn to trust the systems based on their 
experiences. However, trust does not happen overnight; it takes time and follows a 
process of learning (cause and effect).Positive experiences and expectations are needed 
to trust the reliability of these automated machines.For automation technology to work 
as designed, humans must learn to trust it eventually, albeit not blindly. According to 
researchers Clare, Cummings, and Repenning, “Either overtrust or undertrust in 
automation can be detrimental to performance.”*^ 

Overtrust and undertrust also affect designed effectiveness. On the one hand, 
overtrust runs the risk of relying on technology beyond its capabilities. On the other hand, 
the realization of technology effectiveness is not achievable when the user undertrusts 
automation. In other words, automation relies on the user to agree inherently to a certain 
amount of trust in the system. Without this relationship, the design of the technology does 
not result in the benefits it is intended to achieve. What must be noted is the relationship 
can result in the human being conditioned to trust the machine beyond its capability. 

Brown’s fatality is a prime example of the consequences of overtrusting 
technology. Given he did not see the semi truck, it is apparent his attention was elsewhere 

*5 National Highway Traffic Safety Administration, ODI Resume, 1. 

*6 Andrew S. Clare, Mary L. Cummings, and Nelson P. Repenning, “Influencing Trust for Human- 
automation Collaborative Scheduling of Multiple Unmanned Vehicles,” Human Factors 57, no. 7 (2015): 
1208-1218. 

John D. Lee and Katrina A. See, “Trust in Automation: Designing for Appropriate Reliance.” Hfes 
Human Factors: The Journal of the Human Factors and Ergonomics Society 46, no. 1 (2004): 50-80, doi: 
10.1518/hfes.46.1.50.30392. 

*8 Ibid. 

8^ Clare, Cummings, and Repenning, “Influencing Trust for Human-automation Collaborative 
Scheduling of Multiple Unmanned Vehicles,” 1208-1218. 
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and he trusted Autopilot. John D. Lee and Katrina A. See, two experts in human systems 
analysis, claim, “Trust is considered a key variable for reliance on, and misuse/disuse of 
automated sy stems.”^0 

Although Brown’s death is the only reported fatality while using Autopilot, many 
documented cases demonstrate overtrust in the technology. Brown’s case is not an 
anomaly but rather an example of the worst consequences of overtrusting technology. 
Take for instance, the numerous videos online of people showing off their trust in Tesla’s 
Autopilot. 

In one particular YouTube video, a couple demonstrates the trust they have in 
Autopilot by playing cards, playing scrabble, reading, and sword fighting, all while 
Autopilot is driving the vehicle.^i Others have foolishly climbed into the back seat, 
leaving the driver position vacant, with disregard for their safety.92 As ridiculous as these 
drivers appear, it makes the point; trust grows overtime and leads to the slippery slope of 
overreliance. 

Although in these examples the drivers’ trust is more overt given the silliness and 
extreme behavior they demonstrate, it is ultimately the same overtrust Brown 
experienced. The only difference with Brown’s case, assumed to be his inattentiveness, is 
that he was engrossed in an activity that required a lot of his attention, perhaps working 
on his computer or texting on his phone. No one will know for sure; however, what is 
obvious is Brown trusted the Autopilot enough to disengage completely from the role of 
human supervisor. 

Thus, an interesting concern is raised. When should self-driving vehicles be fully 
trusted? Looking at the five levels of automation in Figure 4 helps to identify, in theory, 
when it is appropriate to trust self-driving technology. As noted in the illustration, the 
current level of self-driving technology is level three (indicated with the red star). The 

90 Lee and See, “Trust in Automation: Designing for Appropriate Reliance,” 50-80. 

91 “Autopilot Road Trip,” YouTube video, 0:58, posted by Creative Works, Inc., June 7, 2016, https:// 
www.youtube.com/watch?v=2vNH-Q7fu7A. 

92 “Tesla Autopilot Stunt w/Driver in Back Seat—DO NOT ATTEMPT,” YouTube video, 1:30, 
posted by Jason Hughs, October 16, 2015, http://www.youtube.com/watch?v=-okFVuHlxII. 
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yellow box indicates when self-driving technology will theoretically be capable of all 
tasks a human driver can perform in all environments and road conditions.In essence, 
the human role is simply one of a passenger once self-driving vehicles reach level 5 
automation.94 Before that time, the human must remain a supervisor of the technology. 
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Figure 4. SAE International Levels of Self-driving Technology Modified to 
Explain Appropriate Levels of Trust in the Technology's 


3. Automation Bias: Who Is in Charge? 

Overtrust in automated technology leads to automation bias and affect decision 
making. As defined, automation bias is also known as an, “overreliance on 

93 Bruce, “Collision between a Car Operating with Automated Vehicle Control Systems and a Tractor- 
Semitrailer Truck near Williston, Florida. May 7, 2016.” 

94 Ibid. 

95 Adapted from National Transportation Safety Board, Collision between a Car Operating with 
Automated Vehicle Control Systems and a Tractor-Semitrailer Truck Near Williston, Florida May 7, 2016, 
24. 
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automation.Psychologist Raja Parasuranman helps to understand better the risk 
associated with automation bias by suggesting, “Automation bias can be conceived of as 
a special case of human decision biases, such as confirmation bias and discounting 
bias.”97 Operators use automation cues as heuristics for making decisions. Although the 
use of heuristics is usually effective, occasionally it may lead to error because of decision 
biases.98 Brown demonstrated he understood autopilot’s fallibility and the technology 
was still in the infancy stage of learning, yet was growing more comfortable with the 
autopilot feature, as seen in several of his posts.99 As Brown states, in a November 1, 
2015, post, “Again, this is not an autonomous driving car. It does do amazingly well, 
though. In my supervisory role while on the interstate it rarely requires any input from 
me.” 100 His positive experiences with the technology may be a center point for his 
heuristic development. An article by judgmental forecasting researcher, J. A. Alvarado- 
Valenci suggests, “explanations and past performance are good candidates to increase 
trust in computer’s advice.”ioi The positive experiences support the idea that Brown’s 
past performance and experience, or trial and error, with the self-driving technology 
helped increase his automation bias. 

Several experiences with autopilot helped Brown develop a heuristic about the 
self-driving technology that it was, in fact, safe and reliable. In one particularly powerful 
video, his dash camera reveals autopilot demonstrating its worth by avoiding an accident 
with a large truck on the interstate. Brown’s YouTube video caught the attention of 
Tesla’s CEO Elon Musk. Musk retweeted Brown’s video and received a lot of attention 


96 Parasuraman and Riley, “Humans and Automation: Use, Misuse, Disuse, Abuse,” 249. 

97 Raja Parasuraman and Dietrich H. Manzey, “Complacency and Bias in Human Use of Automation: 
An Attentional Integration,” Human Factors: The Journal of the Human Factors and Ergonomics Society 
52, no. 3 (2010): 405, doi: 10.1177/0018720810376055. 

98 Ibid., 398. 

99 Comment on “Joshua Brown Facebook Post,” Joshua Brown’s Facebook page, November 1, 2015, 
https://www.facebook.com/joshua.brown. 16940/posts/10153614478630734. 

100 Ibid. 

101 “Computing; Studies from Pontifical Catholic University Describe New Findings in Computing 
(Reliance, Trust and Heuristics in Judgmental Forecasting),” Computer Weekly News, August 14, 2014, 
508, http://libproxy.nps.edu/login?url=http://search.proquest.com/docview/1551474305?accountid= 12702. 
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with over three million views. ^02 Musk’s attention to Brown’s post left a lasting 
impression on him. Neither Brown nor Musk understood the confirmation bias that 
presented itself through several interactions. Less than a month before his fatal crash, 
Brown posted, “I can die and go to heaven now! Elon Musk Tweeted about my 
video!”i03 Later in the post. Brown discusses an offer of monetary compensation for the 
rights to his video, which arguably helped further reinforce his heuristic about the 
autopilot capabilities and contributed to his automation bias and a heuristic that helped 
contribute to his death. 1^4 

C. THE BIAS CHALLENGE 

It is a natural tendency for humans to become over reliant on technology. After 
all, technology is meant to make life easier and more efficient but it is a slippery slope. 
For instance, take the following example that demonstrates how humans easily become 
over reliant on technology and fall prey to automation bias. The smartphone features are 
an excellent example to help illustrate this point. Experience tells users the navigation 
system or alarm clocks on smartphones are reliable. Often, a navigation system will 
guides users to their destinations in the most efficient manner possible. Over time, users 
learn these devices are reliable and do what their manufacturers claim they will; earning 
users’ trust even if intuitively humans understand technology sometimes makes mistakes. 
Often unconscious, humans tend to learn in the same manner; therefore, it is likely others 
will fall prey to the same bias as the previous examples demonstrate. In applying the 
same principle of overtrust to self-driving vehicles, drivers may fall victim to the same 
bias, leading to overreliance and eventually automation bias. It is one thing to develop a 
bias towards a morning alarm clock but much more dangerous to develop a bias towards 
self-driving vehicles. 


102 Brown, “Autopilot Saves Model S.” 

Comment on “I Can Die and Go to Heaven Now!,” Joshua Brown’s Facebook page, April 17, 
2016, https://www.facebook.eom/joshua.brown.16940/posts/10153987903875734. 

104 Ibid., April 18, 2016, 7:02 am. 

105 Parasuraman and Riley, “Humans and Automation: Use, Misuse, Disuse, Abuse,” 249. 
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Given the natural tendency for humans to fall prey to these cognitive biases, an 
opportunity exists for the development and design of technology to take these biases into 
consideration and safeguard against them. It is important to understand the different 
heuristics regarding technology and how these ideas relate to humanity. With a better 
understanding of these heuristics, it is then possible to offer possible solutions to the 
human-machine interface. What is interesting is the idea that humans and intelligent 
machines (technology) are separate and must remain separate. This idea does not offer a 
collaborative solution to the problem but continues to foster a division between humans 
and technology and potentiates the frictions and noise at the level of the human-machine 
interface. 
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III. TECHNOLOGY: THE PROMISE AND PERIL 


Technology and its advancements offer great promise for humanity; however, 
some believe technology promises great peril and may outsmart humanity and pose an 
existential threat. This chapter takes a look at models useful in understanding the social 
constructs about the human-technological interface. Among them are different social 
constructs regarding technology and why they are important, how early adopters, such as 
Josh, influence the human-technological interface, the discourse between technology’s 
advancements and humanity’s ability to adapt. Finally, discussed is the doomsday 
prediction of singularity. Since social constructs are created, a unique opportunity exists 
to understand them and influence them. By creating a positive social construct regarding 
the interface between humans and technology, it offers a positive future for humanity. 

A. SOCIAL CONSTRUCTS 

How people come to understand and think about technology is socially 
constructed. Social construct are basically ideas about something that society has decided 
to accept but are not always based on facts. According to Berger and Luckman who 
wrote the book. The Social Construction of Reality, social constructs are important 
because if people can understand how they develop, an opportunity is available to 
influence and change these constructs to best suit the needs and provide for society’s 
safety, 

At the center of social constructs are heuristics. Heuristics give humans a 
“thinking shortcut,” and serve as an expeditious way to make decisions. Without these 
shortcuts, every decision would require time to stop and think about each variable, which 
makes decision making much too slow. Nonetheless, heuristics are usually helpful for 

“Social Constructs,” International Encyclopedia of the Social Sciences, accessed November 18, 
2017, http://www.encyclopedia.com/social-sciences-and-law/sociology-and-social-reform/sociology- 
general-terms-and-concepts/social-3. 

Peter L. Berger and Thomas Luckman, The Social Construction of Reality: A Treatise in the 
Sociology of Knowledge (New York: Doubleday, 1989). 

Doug Bernstein and Peggy Nash, Essentials of Psychology, 4th ed. (Boston: Houghton Mifflin 
Company, 2008), 256. 
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solving problems; they may also “lead to cognitive biases” and errors in thinking, such as 
automation biasd^^ 

Many factors play into developing heuristic and social constructs. One specific 
factor is the early adopters of technology. Josh was an early adopter of Autopilot that 
resulted in a certain mindset that contributed to his social construct of the innovative 
technology in his car. Early adopters believe in the product they have adopted and have a 
vested interest in its success. The early adopter mind set contributes to building a 
heuristic about the technology but also, in Josh’s case, leads to automation bias that 
ultimately contributed to his death. The good news is, if social constructs, such as 
automation bias, are inventions of society, then they can be modified or changed as 
needed. Having a better understanding of how heuristics work and the potential cognitive 
biases they introduce may assist users and developers of technology, and for this thesis, 
self-driving technology, to design and safeguard against human fallibility and biased 
decision making better, 

1. Early Adopters 

Early adopters speed up the testing cycle and reduce time-to-market, and increase 
market share for companies. Everett Rogers coined the term “early adopters” in his 1962 
book. Diffusion of Innovations, which is currently in its fifth edition. The volume remains 
a leading model to help explain how innovations and technology are adopted into 
society, m Early adopters of technology are at the cutting edge of innovation 
development and the human-machine interface, as shown in Eigure 5. Whether 
consciously or unconsciously, ideas and misconceptions affect the adoption and 
perfection of technology. Technology often requires several iterations, and by leveraging 
the early adopters as “guinea pigs,” their feedback is elicited to provide valuable 
feedback to help improve and evolve their products. 


Bernstein and Nash, Essentials of Psychology, 256. 

Kendra Cherry, “How Heuristics Help You Make Quick Decisions,” Verywell, August 15, 2017, 
https://www.verywell.com/what-is-a-heuristic-2795235. 

Everett M. Rogers, Diffusion of Innovations, 5th ed. (Riverside: Free Press, 2003). 
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Early adopters are on the front line of product rollout and function as a feedback 
loop for technology companies in today’s market as they iterated^2 with this 
understanding, many technological products are given to early adopters before they are 
proven safe to do so. As noted in Salim, Malone, and van Geest’s book. Exponential 
Organizations: 

The MVP [Minimum Viable Product], is a kind of applied experiment to 
determine the simplest product that will allow the team to go to market 
and see how users respond (as well as help find investors for the next 
round of development). Feedback loops can then rapidly iterate the 
product to optimize it and drive the feature roadmap of its development. 
Learning, testing assumptions, pivoting and iterating are key in this 
step.”' 

What used to take years of research, design, and testing is now accomplished by 
releasing products before they are statistically reliable to allow the end users experience 
with the technology to shape the product. Today, it is considered crucial for end user 
feedback to “get the product right.”Functional technology is very important for user 
satisfaction but also brings substantial monetary gains for companies. Since the market 
share of companies’ product is one of the most critical metrics for business success, 
speeding up the process of research and design saves a lot of money and ensures greater 
market. 116 Unfortunately, not all parties involved benefit equally. Reducing cycle time 
serves companies, builds competitive barriers, but may also endanger the early adopters. 
Roger’s explains the early adopters are key to introducing products to the market. In 
essence, early adopters facilitate a tipping point or a place that creates a domino effect 
with society that greatly improves its market share (see yellow line in Figure 5).ii^ 


112 Salim Ismail, Michael S. Malone, and Yuri Van Geest, Exponential Organizations: Why New 
Organizations Are Ten Times Better, Faster, and Cheaper than Yours (and What to Do about It) (New 
York: Diversion Books, 2014), loc. 2317 of 4417, Kindle. 

113 Rogers, Diffusion of Innovations. 

114 Feedback loops can then rapidly iterate the product to optimize it and drive the feature roadmap of 
its development. Learning, testing assumptions, pivoting, and iterating are key in this step. 

113 Ismail, Malone, and Van Geest, Exponential Organizations: Why New Organizations Are Ten 
Times Better, Faster, and Cheaper than Yours (and What to Do about It, loc. 2317-2318. 

116 “Definition of ‘Market Share,’” Economic Times, accessed November 8, 2017, https://econ 
omictimes.indiatimes.com/definition/market-share. 

112 Rogers, Diffusion of Innovations. 
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Figure 5. Everett M. Rogers’ Diffusion of Innovation and Cycle of Adoption 
(Blue Line) in Comparison with Market Saturation (Yellow Line)ii* 

While some technology is not dangerous when in its initial phases of 
development, I argue the self-driving vehicle is quite a different story. Much more is on 
the line for the earliest of adopters who are beta testers for self-driving vehicles than say 
the latest version of Google docs or the latest version of an Android phone. All the while, 
companies save thousands, if not millions of dollars by putting their product out quickly 
and make iterative changes as feedback cycles in from the end user. The greater the 
market share for companies who develop this technology first, the harder it is for other 
companies to enter the market creating a foothold for them among customers. Early 
adopters face another challenge they are often unaware of, that of their own bias. 


Source; Ibid. The graph was not modified but may have some errors, as 100% of market share is 
highly unlikely and there is nothing greater than 100%, as indicated by the arrow on the y-axis. 

ll^Maik T. Schneider, Burkhard Schade, and Hariolf Grupp, “Innovation Process ‘Fuel Cell Vehicle’; 
What Strategy Promises to be Most Successful?” Technology Analysis & Strategic Management 16, no. 2 
(2004); 152, 171, doi; 10.1080/09537320410001682874. 
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2. Analysis of Brown’s Automation Bias 

Although early adopters may be the safer operators of these vehicles because they 
inherently understand the vehicles are in their first iteration, I contend the opposite is true 
and early adopters are likely unaware of their bias, as they have a vested interest in seeing 
the technology being successful. It may be more accurate to say these early adopters are 
better described as “early believers.” 

Josh was an early believer and his belief helped build a heuristic about the 
vehicle’s capabilities. Josh’s early adopter mindset put him at greater risk for injury, and 
in his case, death. Being one of the earliest adopters of self-driving technology, how he 
came to think about himself and technology was socially constructed. The analysis from 
Chapter II helps to explain how trust, ideas, and misconceptions influenced a growing 
trust with the technology, influenced his decisions, and led to automation bias. 

3. Teller’s Model and Humanity’s Adaptability 

One heuristic about the human-technology interface comes from three-time 
Pulitzer Prize winner, Thomas Friedman. Friedman offers a framework from which to 
look at the relationship between humans and technology. ^ 20 fjg believes humans are not 
capable of catching up to technology’s rapid advancements. While having lunch with his 
friend, Eric “Astro” Teller [Google X Research and development lab], also known as the 
“Captain of Moonshots,” Teller drew a sketch for Friedman to help explain the discourse 
between humans and technology (see Figure xhe bottom line he claims is humans 
are struggling to keep up with technology’s advancing rate of change. 

In his book. Thank You for Being Late, Friedman uses Teller’s graph to illustrate 
several reasons for the shift including why humans are not keeping up with technological 
changes. Technology advancements have taken place for decades; however, in 2007, the 
world saw an exponential change or hockey stick moment for technology. 122 On the 
graph, the line representing technology starts out with a gradual climb but right around 

120 Friedman, Thank You for Being Late: An Optimists Guide to Thriving in the Age of Accelerations. 

121 Ibid. 

122 Ibid. 
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2007, a steep almost vertical upward curve occurredd23 The line representing humanity’s 
ability to adapt looks very different, almost flatd24 It does gently rise over time but given 
the trajectory of each line, the gap between the two will only continue to widen. 



What happened in 2007 to cause such a severe change? Several famous releases 
in technology occurred during that year. Steve Jobs introduced a revolutionary new 
communication platform, the iPhone. 1^6 Android released its phone around the same 
time, and shortly afterwards, the soon to be giant, Facebook, went public. The 
smartphone allowed humans to connect directly with the Internet and brought the 
“computing power of the Apollo Space Mission,” to the palm of their hands. 1^8 These 


123 Friedman, Thank You for Being Late: An Optimists Guide to Thriving in the Age of Accelerations. 

124 Ibid. 

125 Source; Ibid., 34. 

126 “Thomas L. Friedman; “Thank You for Being Late,”“ YouTube video, 11;54, posted by Talks at 
Google, February 22, 2017, https;//youtu.be/nuF2JKeM2CY. 

127 Ibid. 

128 Ibid. 
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releases would revolutionize how people communicate and connect with one another on a 
worldwide sc ale 

Why does technology progress at such a fast rate? The answer dates back to 1965 
and the microchip. Another evolution, and perhaps the most instrumental for advancing 
technology, started in 1965. The microchip is perhaps the most instrumental reason for 
the rapid advancement of technology. In 1968, with a PhD in chemistry and physics from 
Caltech, Gordon Moore cofounded Intel, which is worth over eight billion dollars as of 
October 2017, and is considered one of the fathers of Silicon Valley.Naming it after 
himself, Moore predicted that the microchip would allow computing power to double 
every few years. American physicist, James R. Powell explains Moore’s law in greater 
detail. He states, “Moore’s prediction is that the density of transistors and computing 
power doubles every two years, which has held since there were fewer than 100 
transistors in an integrated circuit until today with many millions of transistors on a single 
integrated computer chip. ”132 xhe recent switch from silicon to non-silicon material in 
microchips allowed Intel technology companies to continue to produce and release new 
products at an unprecedented rate.i33 Moore’s Law allows technology to change 
exponentially and this trend continues today. 134 The processing ability of today’s 
computers facilitates the hockey stick moment witnessed in 2007.135 

In applying Teller’s theory to the rapid development of self-driving technology, 
understanding the current transition is a must. As self-driving technology begins to 
integrate, the relationship between humans and technology is changing and will affect 
outcomes. According to the NTHSA, “New complexity is introduced as HAVs [highly 

129 “Thomas L. Friedman; “Thank You for Being Late,”“ 12:04. 

130 “Gordon Moore,” Forbes, accessed October 26, 2017, https://www.forbes.com/profile/gordon- 
moore/. 

131 Ibid. 

132 James R. Powell, “The Quantum Limit to Moores Law,” in Proceedings of the IEEE 96, no. 8 
(2008): 1247, doi: 10.1109/jproc.2008.925411. 

133 Friedman, Thank You for Being Late: An Optimists Guide to Thriving in the Age of Accelerations, 
19-20. 

134 Ibid., 19-21. 

135 Ibid. 
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automated vehicles, what this thesis calls self-driving vehicles,] take on driving 
functions. ”136 Early adopters are at the heart of moving technology forward but they are 
at great risk in this transition time. When a mix of vehicles driven by both humans and 
computers are sharing the road, it “reshapes” how humans see their relationship with 
vehicles. 

The early adopter may be more likely to experience automation bias. Early 
adopters need to be more vigilant and understand their tendency to trust technology 
especially when it is in its first generation. Until technology proves to be as reliable in an 
array of situations, early adopters should prepare for and expect technology to fail or 
cause errors. Automation bias is especially dangerous when the technology is in its 
infancy. 137 Eor the remainder of this thesis, I call this transitional time, the “hybrid 
phase” (Figure 7). 


136 “Federal Automated Vehicles Policy—September 2016,” Department of Transportation, 

September 19, 2016, 22, https://www.transportation.gov/AV/federal-automated-vehicles-policy-september- 
2016/. 

137 “Driver Errors, Overreliance on Automation, Lack of Safeguards, Led to Latal Tesla Crash.” 
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Figure 7. Astro Teller’s Model Modified to Explain the Hybrid Phase: Humans 

and Computers Driving Vehiclesi^s 

4. Modifying Teller’s Model to Explain the Hybrid Phase of Self-driving 
Vehicles 

During the hybrid phase of self-driving vehicles, the role of a human driver begins 
to shift from operator to a more passive supervisor. The skill sets necessary to drive 
legacy vehicles does not automatically transfer to the self-driving vehicles. Thus, safety 
concerns are created that are not intuitive; therefore, it should not be overlooked. Going 
from operator to supervisor is a big change and causes disorientation if the human needs 
to transition quickly to the operator role. 

Although only a small percentage of society’s vehicles have self-driving 
technology, the hybrid phase should concern anyone who cares about safety while 
driving. Toggling between vehicles with advanced driver assist features (not quite self¬ 
driving), such as those with backup cameras, and legacy vehicles, is problematic for early 
adopters and drivers. Greater dependence on technology and more advanced features may 
disorient drivers upon switching to using a vehicle without such luxuries. 

Adapted from Friedman, Thank You for Being Late: An Optimists Guide to Thriving in the Age of 
Accelerations, 34. 
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While self-driving technology is intended to enhance and improve driver skill 
sets, unfortunately, humans tend to adapt and rely on technology, often unconsciously. 
Reliability can cause a change in behavior. The behaviors that usually keep people safe 
but change due to reliability creates a safety concern. Habitual behaviors, for instance, 
shoulder checking before a lane change, are being replaced with cues from the vehicle 
computer. If sensors detect a vehicle in a blind spot, the computer illuminates a small 
light in the side mirror that indicates the danger. 139 

When humans become more reliant on assists, they begin to lose their skills sets 
in these areas and form new habits. As long as the vehicle’s computer is working 
correctly, all is well; however, if a driver begins to rely on the technology and that 
technology fails, the habitual behavior is no longer readily available and causes 
disorientation and increases risk. For example, if a driver starts to enter an adjacent lane, 
fails to shoulder check, does not see a car, and the vehicle does not work as designed—no 
illumination of the light in the mirror—the potential for a collision increases. In this 
example, something designed to be helpful and improve safety may actually condition the 
driver to be dependent upon the technology. 

The same argument can be made with vehicles that drive themselves. As drivers 
use autopilot features more and more, their dependence on them increases. Dependence 
increases risk, particularly if technology fails or the design is flawed. The potential for 
danger during the hybrid phase cannot be understated. Technology is still developing and 
humans are learning their new role with advancing technology.in the case of the early 
adopters, the stakes do not get any higher, which makes this case study an incredibly 
important warning; the hybrid phase of self-driving technology is a time for increased 
awareness and caution. 

Technology’s reliability will obviously play a significant role in risk, which raises 
the question of what is an appropriate level of reliability for self-driving technology? It is 

139 “Blind Spot Information System,” Ford, accessed November 3, 2017, https://www.ford.com/cars/ 
mustang/2017/features/technology/. 

National Transportation Safety Board, Collision between a Car Operating with Automated Vehicle 
Control Systems and a Tractor-Semitrailer Truck Near Williston, Florida May 7, 2016, 41^2. 
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likely to take some time before self-driving technology is statistically reliable; whatever 
that means. Until that time, the hybrid phase presents dependency and vulnerability issues 
that are not understood. 

Remembering back to earlier in the chapter, Teller’s model offers one explanation 
for the gap between technology’s rate of change and humanity’s inability to adapt fast 
enough. 141 Friedman primarily uses this model to illustrate this disproportioned 
relationship. He also explains how the “‘machine’ reshape parts of our society such as, 
politics, the geopolitics, ethics, the workplace, and community. ”142 xhis idea is certainly 
an interesting concept and should be taken into consideration. Is the machine affecting 
ethics in technology’s design? Whoever develops the bona fide self-driving vehicle first 
will likely make a lot of money, which creates a wickedly competitive landscape. 
Competition is good and bad because it drives forward progression but it may also cut 
comers to get ahead. The next section warns of forward progress and the risk of getting 
too far ahead. Ensuring safety above all else must remain an ethical benchmark. 

B. SINGULARITY (AKA DOOMSDAY) 

As mentioned earlier, according to Berger and Luckman, social constmcts are 
important because by understanding how they develop, an opportunity exists to influence 
and change them to best suit the needs and society’s safety.143 One particularly 
controversial social construct is the idea of a future of singularity. This idea proposes that 
humanity is on an inevitable trajectory where AI technology will pose an existential 
threat to humanity because its capability will outstrip those of humans. Many scientists 
and respected thinkers share this viewpoint. Brostrom and Hawking alike assume that AI 
will take away humanity’s ability to create its own future; however. Hawking does offer a 
way out. He believes if humans can implement safeguards now, the inevitable fate of 
humanity can be avoided. 

^^41 National Transportation Safety Board, Collision between a Car Operating with Automated Vehicle 
Control Systems and a Tractor-Semitrailer Truck Near Williston, Florida May 7, 2016, 41^2. 

142 “Thomas L. Friedman; “Thank You for Being Late,”“ 8;50. 

^43 Berger and Luckman, The Social Construction of Reality: A Treatise in the Sociology of 
Knowledge. 
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Hawking and Bostrom comment that a techno-centric approach to intelligent 
agents will result in a techno-centric world that excludes humanity If an accurate 
assumption, then the opposite is also true. By developing artificial agents but ensuring a 
human-centric approach, humanity can ensure a human-centric rather than an inevitable 
techno-centric world. Humans should realize they are powerful independent thinkers and 
are not the simple sum of brain cells, as inferred by the singularity debate. Humans are 
incredibly complex and hold the ability to influence their future. Blyth et al. in their 2015 
paper, “Driving the Self-Driving Vehicle: Expanding the Technological Design Horizon, 
support the idea that the future is not “inevitable,” but rather, something to be shaped. 1^5 
As Blyth concludes, “the future is something that can be shaped, rather than being 
already decided or “inevitable.” 1^6 

Learning what human-centric means in the digital era requires definition and 
goals to make it happen. Humans have an uncanny ability to create that on which they 
focus their attention. I propose that humans have a vested interested in seeing a future in 
which artificial agents assist or augment humans but do not take over. Self-driving 
vehicles are perhaps the first opportunity for humans to get the future right. By 
leveraging the power of technology while maintain a human-centric approach to 
development and design, a synergistic relationship will emerge and may offer a quantum 
leap in evolution for mankind. 

Experts and developers believe connected self-driving technology will greatly 
improve highway safety. Considering that approximately 35,000 Americans are killed on 
the roads every year, solving this problem will have a significant and positive impact on 
society. 147 The NTSB recently recommended that upon completion of connected vehicle 


144 Hawking, “Stephen Hawking: ‘Transcendence Looks at the Implications of Artificial 
Intelligence—but Are We Taking AI Seriously Enough?’”; Bostrom, Superintelligence: Paths, Dangers, 
Strategies. 

145 Pascale-L. Blyth et al., “Driving the Self-driving Vehicle: Expanding the Technological Design 
Horizon,” in Proceedings of the Technology and Society (ISTAS), 2015 IEEE International Symposium on 
2016-March Institute of Electrical and Electronics Engineers Inc., 201, 7439419 (2016): 1-6, doi: 10.1109/ 
istas.2015.7439419. 

146 Ibid. 

147 “Pederal Automated Vehicles Policy—September 2016,” 5. 
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technology meeting the minimum performance standards, the NHTSA should require that 
all new highway vehicles have this technology installedd^^ The Chairman of the House 
Subcommittee on Digital Commerce and Consumer Protection, Bob Latta, believes self¬ 
driving vehicles have the “potential to transform our transportation system into one that is 
safer and more secure for everyone on the roadways.”i49 

Of course, many will disagree with this assertion that self-driving technology 
significantly improves driving safetyEven if future regulations require equipping new 
vehicles with self-driving technology, vast amounts of vehicles will remain on the road 
with little or no connectivity. Transitioning older vehicles off the roads could take 
decades, especially considering non-believers, as well as driving enthusiasts, may refuse 
to transition from their legacy vehicles to vehicles with self-driving technology. 
Therefore, the shift to self-driving vehicles will take time. Until then, this nation is in a 
time of transition and may not see as much of a benefit as expected, 

At this juncture, ethics re-enters the narrative. The ethics of ensuring the human 
remains the center focus for all technology may offer an alternative future. I believe 
humans do have a say in what happens in the future; however, a different social 
commentary on this topic is needed to ensure humans have a say. In the next chapter, I 
suggest an alternative social construct around the integration of humans and technology. 


National Transportation Safety Board, Public Meeting of September 12, 2017, Collision between a 
Car Operating with Automated Vehicle Control Systems and a Tractor-Semitrailer Truck, Williston, FL, 
May 7, 2016, NTSB/HAR-17-XX (Washington, DC: National Transportation Safety Board, 2017), abstract, 
https://www.ntsb.gov/news/events/Documents/2017-HWY16FH018-BMG-abstract.pdf. 

^^49 Subcommittee on Digital Commerce, House of Representatives, 115th Cong., 1st. sess., February 
14, 2017. 

150 Without straying too far from the primary discussion in this chapter, it is worth noting that not all 
experts agree that self-driving vehicles are safer. In fact, some make the case that cyber-security poses a 
significant concern about connected vehicles. Consider what could happen if someone hacked self-driving 
vehicles and abruptly made all vehicles take a sharp left turn. Cyber-security safeguards should be a topic 
that is taken seriously. 

National Transportation Safety Board, Collision between a Car Operating with Automated Vehicle 
Control Systems and a Tractor-Semitrailer Truck near Williston, Florida, May 7, 2016, 13, finding. 
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IV. BROWN’S POINT: SUCCESSFUL ADAPTATION OF 

TECHNOLOGY 


Thus, far, using Brown’s fatality as an unprecedented case study offers an 
opportunity to understand the interface and human and technological fallibility. 

A. BROWN’S POINT: AN ALTERNATIVE SOCIAL CONSTRUCT 

In this chapter, I propose an alternative social construct to the singularity 
prediction and call it Brown’s Point. Intended as a way forward for the interface between 
humans and technological machines, it also serves as a countemarrative to the doom and 
gloom of the singularity theory. It serves as a tipping point and positive beacon for 
humanity that puts the human at the center of the human-technology relationship by 
combining the best of both technology and humanity. 1^2 xhis synergistic relationship is 
achieved when technology augments humans and fills the gaps created by human biases 
that can take humanity to the next level of thinking. Brown’s Point is where the human- 
technological interface reaches the beginning of its potential. 

Brown’s Point is a moment in time when humans fully adopt technology and 
work together at an optimal level, as shown in Figure 8. Brown’s Point creates more than 
an additive potential; it creates synergy. This synergy is likely only to happen when 
technology becomes reliable with the appropriate level of market saturation (adoption) 
and humans have adapted to utilize the technology. The partnering of humans with 
advancing technology is likely to enhance human capability greatly. Friedman believes 
the advancing technology is in essence a supernova for advancement. 1^3 If true, it surely 
happens when Brown’s Point is reached. 


^32 Rogers, Diffusion of Innovations. 

153 Friedman, Thank You for Being Late: An Optimists Guide to Thriving in the Age of Accelerations. 
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Safety 



Time 


Figure 8. Brown’s Point: The Optimum Interfaee between Humans and 

Technology 


The smartphone may be an example of a technology used by humanity that has 
successfully reached Brown’s Point. The enhancement for humanity with mobile 
computing power brings exponential ability to process information faster, communicate 
easier, and flatten the world. Friedman’s use of Teller’s chart to illustrate technology’s 
rate of change shows the graph swung drastically upward around 2007; however, this 
change is only part of what was really happening in 2007. What Friedman and Teller fail 
to recognize is the line representing the human should also take a drastic upswing on the 
graph. Smartphones have become a part of daily life for the majority of society. Due to 
the partnership seen with smartphones, the hybrid time decreases and Brown’s Point of 
exponential potential is realized. The smartphone in essence is now a one-stop-shop. 
What used to require several devices is now available in one. Phones, cameras, 
computers, alarms, reminder lists, portable digital optical disc (DVD) players, live video 
streaming, crowd sources applications, social media access, and the list goes on, come in 
the tidy smartphone package. The efficiency and access to information and 
communication for humans with this technology in the palm of their hands creates an 
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exponential capability. The synergistic relationship between smartphone technology and 
humans serves as an example of what is possible when Brown’s Point is reached. 

Self-driving technology offers the same opportunity. When self-driving vehicles 
attain saturation (market share), it will be an enormous step forward for the safety of U.S. 
highways, connectivity with infrastructure, as well as autonomy for many people who 
otherwise cannot drive. When vehicles connect in the future, many things are likely to 
change from how goods and services are moved, to improving the congestion on the 
roads in addition to enhanced efficiency and convenience. 1^4 With such promise, 
ensuring self-driving vehicles reach Brown’s Point is worth the effort. 

So how can Brown’s Point be accomplished? As mentioned in Chapter I, self¬ 
driving vehicles represent three variables: technological advancement, human’s ability to 
adapt, and the interface between technology and adaptation. With the mastery of these 
three variables, reaching Brown’s Point is possible. 

B. GETTING TO BROWN’S POINT: SHORTENING THE HYBRID PHASE 

The hybrid phase is at the center of a hurdle for self-driving technology to 
navigate before it is capable of reaching Brown’s Point. Therefore, shortening the time in 
the hybrid period allows technology to reach Brown’s Point quicker. I offer a few ways to 
shorten the hybrid phase of technological integration. One is to slow the advancement of 
technology, thereby allowing humans to catch up. The second way is to improve the 
reliability of technology so humans will adapt more easily and adopt them more quickly. 
Finally, increase humanity’s ability to adapt and adopt self-driving technology. 

Slowing technology’s progression seems like the easier of the two fixes. To see 
the greatest potential and have true synergy, technology and humanity need to be 
performing at their best. Finding a way to allow technology to advance safely and for 
innovation to thrive while ensuring humans advance together with the technology is the 
most important piece in reaching Brown’s Point. 

^^54 James Schulte, “U.S. DOT Releases New Automated Driving Systems Guidance,” National 
Highway Traffic Safety Administration, October 27, 2017, https://www.nhtsa.gov/press-releases/us-dot- 
releases-new-automated-driving-systems-guidance. 
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Considering that technology is more efficient and more capable of processing 
information and solving problems than the human in many respects, machine learning 
offers a tremendous benefit for humansd^S As mentioned in Lance Whitney’s Time 
magazine article, “computers are simply more accurate at pulling off a broadening range 
of high-value functions than we are. They’re not affected or influenced by emotions, 
feelings, wants, needs and other factors [biases].”1^6 when combined with human’s 
intelligent technology, it offers a significant enhancement to human cognition and 
decision making, which is only possible, however, if the technology is programmed 
appropriately. Machines will do what humans ask of them. It is, therefore, incredibly 
important to get the programming right. 

1. Helping Advancing Technology Reach Brown’s Point 

At its most advanced point, self-driving vehicles must reach level 5 automation, 
as shown in Figure 4. The technology is not yet there, and it is hard to predict when the 
technology will reach this benchmark; therefore, the focus must be placed on the here and 
now, or at levels 2 and 3. Humans remain partly responsible for driving—also known as 
the hybrid phase—so technology must partner with humans in the safest way possible. 

Brown’s fatality exposed that both self-driving technology and humans are 
fallible. If technology is programmed to learn and understand human biases, it can offer 
alternative suggestions for the human when making decisions and improving overall 
safety. By leveraging intelligent technology in this manner, the machine has an 
opportunity to assist the human in the most efficient manner. Learning how the driver 
learns, his tendencies, biases, and decision-making processes, coupled with what the self¬ 
driving vehicle senses in the environment, can offer a more global view. The computer 
can then warn the human of an increased amount of risk. By partnering with humans, the 
self-driving vehicle becomes a team player and can offer alternative choices of action to 
counter poor decisions or biases, if needed. 


155 Whitney, “Are Computers Already Smarter Than Humans?” 
Ibid. 
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Regulation may provide a safer way forward for advancing technology but 
technology often outpaces regulationd^^ Hawking and Musk make the argument that 
intelligent agents, such as AI, need regulation to get the future right and not allow the 
idea of singularity to become realityAlthough regulation might be a popular solution, 
it is not as simple as it soundsd^^ Many factors must be considered when implementing 
regulations. Several agencies, including the Rand Corporation, are working on guiding 
documents for policymakers to regulate self-driving technology. ^Further study is 
needed to understand the intended consequences, as well as unintended consequences of 
regulation. 

2. Training Humans to Reach Brown’s Point 

Helping humans adapt to self-driving vehicles is easily overlooked and requires a 
different type of training. The assumption, at least in the Autopilot case, is that skill sets 
will transfer from years of driving legacy vehicles. I propose that assumption is wrong. 
Humans have always thought of themselves as the decision maker and the one 
responsible for awareness and safety in the human-machine interface. Self-driving 
vehicles are changing and quite possibly creating confusion for who is responsible for 
safety and awareness. 

Brown’s case demonstrated the risk of overreliance and automation bias. Training 
the human as a supervisor of self-driving technology may offer a promising solution to 
this problem. Teaching humans what they need to know about the capabilities and 
limitation of the technology is important for the safety of the driver and others on the 
road. Tesla offered some warning of the limitations; however, I would argue writing it in 
a manual without hands-on training and education is insufficient; especially since it is 

Bomey and Zambito, “Regulators Scramble to Stay Ahead of Self-driving Cars.” 

Hawking, “Stephen Hawking: ‘Transcendence Looks at the Implications of Artificial 
Intelligence—but Are We Taking AI Seriously Enough?’”; Tia Ghose, “Elon Musk: Regulate AI before 
Robots Start ‘Killing People,’” LiveScience, July 17, 2017, https://www.livescience.com/59826-elon- 
musk-wants-ai-regulated.html. 

Almost half of the U.S. states have adopted or passed regulation for self-driving vehicles. In 
thinking about the future, it is prudent to create continuity in regulation across the nation. 

James M. Anderson et ak, Autonomous Vehicle Technology: A Guide for Policymakers (Santa 
Monica: RAND, 2016). 
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such a natural human tendency to become overreliant on teehnology. The NTSB 
published this statement in its aeeident report regarding the soft eonstraints that Tesla 
provides for the owners of its vehicles: 

Soft Constraints. As a soft constraint to Autopilot use, Tesla provided 
written instructions in its owner’s manual about those types of roads on 
whieh Autopilot should and should not be used (Tesla 2016). The Tesla 
Model S Owner’s Manual stated that “Traffie-Aware Cruise Control is 
primarily intended for driving on dry, straight roads, such as highways and 
freeways” (p. 68). The manual also provided the following statement: 
“Warning: Do not use Traffie-Aware Cruise Control on eity streets or on 
roads where traffic conditions are eonstantly ehanging and where bieyeles 
and pedestrians are present” (p. 68). Similarly, with respect to the 
Autosteer system, the manual stated, “Warning: Autosteer is intended for 
use only on highways and limited-aeeess roads with a fully attentive 
driver” (p. 74). In diseussing restrieted roads, the manual stated that 
“Autosteer is intended for use on freeways and highways where access is 
limited by entry and exit ramps” (p. 75). The manual also stated that 
“Autosteer is a hands-on feature. You must keep your hands on the 
steering wheel at all times” (p. 74).i6i 

By just looking at his last seetion of the NTSB report, it is easy to see the page 
numbers indieate the owner of the vehicle must have read through 68 plus pages to get to 
this vital safety information. Having it buried in a document that is likely overwhelming 
just given the sophistication of the vehicle, is not a very efficient or safe way to 
communicate. How many people actually sit down and read a manual; I propose very 
few. 


Sinee the self-driving technology is the greatest ehange in vehicles since the car 
was invented, it is prudent to consider re-training drivers in their new tasks. Further, 
drivers need to understand responsibilities and cognitive biases better, and make the 
training mandatory and hands-on. Parasuraman suggests, “training is required to 
recognize and counter deeision biases that may lead to overrelianee on automation.”i62 


National Transportation Safety Board, Collision between a Car Operating with Automated Vehicle 
Control Systems and a Tractor-Semitrailer Truck Near Williston, Florida May 7, 2016, 13. 

“In Memoriam: Noted Researcher and Scholar Raja Parasuraman”; Parasuraman and Riley, 
“Humans and Automation; Use, Misuse, Disuse, Abuse,” 249. 
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With humans meeting machines, the responsibilities of the driver changes and further 
work on training owners is prudent. 

Prior to Brown’s death, Tesla owners had no “required” training. In retrospect, it 
is fairly easy to see the skill set of driving legacy vehicles is not comparable to the role of 
a human supervising a vehicle with Autopilot or self-driving features. Professor Nadine 
Sarter with the University of Michigan specializes in cognitive engineering and argues, 
“training the human in this supervisory role is necessary. 163 The human must learn and 
become an expert who understands the automated system’s capabilities and limitations. 

3. Improving the Interface to Brown’s Point 

Improving the advancing technology and ensuring critical variables, such as 
human bias are accounted for, and improving humanity’s ability to adapt to self-driving 
vehicles, will improve the interface. It is hard to say with certainty what the interface will 
look like in the future, especially considering how iterative a process it is to make 
improvements to technology. The future may see an interface that looks very different 
than what is seen today. Given how dynamic technology is, staying ahead or abreast with 
these challenges is important. Creating flexible and agile processes to analyze and 
quickly make improvements is needed. Research on this unprecedented interface must 
continue and should be a priority for all stakeholders. 


163 Wesley A. Olson and Nadine B. Sarter, “Management by Consent in Human-machine Systems: 
When and Why It Break Down,” Human Factors 43, no. 2 (2001): 255-66. 
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V. ALTERING THE VARIABLES WITH BROWN’S POINT 


Understanding and changing the variables, heuristics, biases, and neuroscience 
associated with the human-machine interface can result in better outcomes. Consider this 
rewritten account of Brown’s accident by altering the variables that contributed to his 
death. 

It is a dry and clear Florida day on May 7, 2016. Joshua just finished a vacation 
with his family at Disneyland. It was the first time in a long time they had all been 
together as a family. Amanda (Josh’s only sister) and her family left Orlando first. The 
kids had a blast hanging out with everyone, especially Uncle Josh; he always knew how 
to facilitate awesome adventures. Josh joined his parents at their campground for a little 
extra time with them. It was not often he had time off as the owner and workhorse behind 
his successful tech-business. He packed up Tessy (the name of his car), embraced his 
Mom and Dad, and got on the road. 

Josh set his adaptive cruise control to nine mph over the speed limit, as he always 
did. With 25,000 miles in only 10 months under his belt driving with the Autopilot 
software upgrade, he knew the car worked well. Even Elon Musk was impressed by a 
video he had posted only a few weeks prior. Autopilot may very well have saved his life 
when he swerved to avoid a collision with a truck on a busy interstate. 

He was now in supervisor mode. With his understanding that he had a tendency 
to rely on technology, he knew he was vulnerable. Even with the capabilities and 
confirmations that Autopilot was working as designed, he keeps his eyes up and his 
attention is on the road ahead. Continuing to scan ahead as taught when learning to drive 
the legacy vehicles of the past, he marveled in how relaxing driving was now that 
Autopilot was at the helm. 

The next leg of his trip was long and straight with few cars on the road with him. 
Although the human mind might fall victim to letting its guard down, the Autopilot 
would warn him that the environment had an increased risk of inattentiveness. This risk 
reinforced the need for Josh to be an active supervisor; a lesson learned in the additional 
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training required to own a self-driving vehicle. The beauty in the Autopilot design is the 
technology helps the human shape an understanding of inattentiveness. New recently 
developed algorithms combine information from the car’s sensors—that observe the 
external environment—and combine them with human biases and tendencies the vehicle 
learns from interacting with its human. The new human-centric technology helps people 
think about their thinking and guide them to better decisions. 

Stopping in Williston, FL, Josh charged up the car and would be good for another 
200 miles or so. Merging onto eastbound U.S. 27, he again set his Adaptive cruise control 
to 74 mph. He crested a small hill and noticed a semi truck in the left turn lane traveling 
in the opposite direction. He always travelled in the right lane unless he was passing. He 
noticed a small convenience store off to the right of the intersection and how green the 
fields were surrounding them. Less than 500 feet from the intersection, he sees the semi 
truck enter the uncontrolled intersection. Thinking to himself, what is this guy doing; 
does he not see me? 

Confident that the Autopilot is aware of the peril, he knows the car will trigger the 
automatic braking system. Josh covers the brake with his foot just in case. To his 
surprise, the car does not slow down; that is weird, he thinks. He applies the brake, which 
automatically turns off the cruise control and puts it back into manual mode. He is always 
so impressed at how fast electric cars can brake. He quickly moves to the left lane and 
barely misses the back of the semi. He lays on the car’s horn hopefully to make the truck 
driver aware of his surroundings. Returning to the right lane, he resets his car to 74 mph. 
In the rear view mirror. Josh watches as the semi truck exits the intersection behind him 
thinking to himself, those semis really need to have self-driving technology installed; that 
was close! 

Josh’s story re-written is important because if offers a window into what is 
possible when a more human-centric approach is taken when combining humans and 
machines. When technology accounts for human bias, it decreases the gap between 
technology and human fallibility, and thus, improves safety. Helping humans make safer 
decisions will likely improve adaptability and adoption of this technology, ultimately 
getting to Brown’s Point. 
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Without careful consideration in combining humans with machines, the risk of 
error increases simply because both are fallible. A lack of continuity in this process 
creates disruption and mistakes. 1^4 When humans only supervise a self-driving vehicle, 
the execution of the decision making while driving can prevent them from developing a 
good understanding of the situation, which is essential to safety. ^65 The more humans 
learn to rely on automation, the more difficult it will become for them to act accordingly 
in an emergency if they need to respond. As humans become more reliant on smart 
machines, the machines themselves must fill the gap for humans and create a positive 
interface. 

Research from the airline industry offers some lessons learned from the human- 
machine interface.Transcontinental pilots utilizing Autopilot features to decrease 
fatigue on long flights tested much lower in the motor skills needed to fly a plane 
manually compared to pilots who only used autopilot for a short time. 1^7 As the auto 
industry becomes increasingly similar to the airline industry, lessons from the aviation 
industry and National Aeronautics and Space Administration’s (NASA’s) automation in 
the cockpit may prove prudent for research and designers developing self-driving 
vehicles. 

Josh’s alternative narrative is possible if the warnings are heeded. The results 
albeit, fictional, offer an alternative to his tragic and avoidable death. Demonstrating the 
benefits of intentional design with the human remaining at the center may save others 
from injury or death in the future, as humans inevitably interface with intelligent 
machines. Designing with intent to ensure machines’ primary directive is to help humans 
on all levels is crucial. ^69 To see the exponential potential of the human-machine 

164 Breton and Bosse, “The Cognitive Costs and Benefits of Automation.” 

165 Ibid. 

166 Andreas Haslbeck and Hans J. Hoermann, “Flying the Needles: Flight Deck Automation Erodes 
Fine-Motor Flying Skills among Airline Pilots,” Human Factors: The Journal of the Human Factors and 
Ergonomics Society 58, no. 4 (June 13, 2016): 533, doi: 10.1177/0018720816640394. 

167 Ibid., 534. 

168 John Pavlus, “What NASA Could Teach Tesla about Autopilot’s Limits,” Scientific American, July 
18, 2016. http://www.scientificamerican.com/article/what-nasa-could-teach-tesla-about-autopilot slimits/. 

169 Whitney, “Are Computers Already Smarter Than Humans?” 
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interface truly, reaching Brown’s Point should be a goal of all advancing technology 
meant to work closely with humans. 

A. FURTHER AREAS OF STUDY AND MY RECOMMENDATIONS 

Due to the infancy of self-driving vehicles, look to the airline industry for lessons 
learned regarding the human-machine interface. 

1. Take a greater human-centric approach to intelligent agent design. Perhaps 
partner with cognitive neuroscientists to understand human bias and 
decision-making processes better during the different levels of self-driving 
technology and test for cognitive biases and automation bias. 

2. Further define what human centric means in the digital era. 

3. Align benchmarks and goals with the different phases of self-driving 
vehicle capabilities to optimize safety. Each phase is different and brings 
different challenges. 

4. Understand how to program technology better to account for human biases 
to reach Brown’s Point. 

5. Regulate self-driving vehicles but ensure the end goal of Brown’s Point. 

6. Develop a social commentary and heuristic that allows the human to 
remain centric in human-machine interface. 

7. Require supervisor training for all self-driving vehicle owners separate 
from the driving education for legacy vehicles. 

8. Study the interface and how it changes with new advancing technology. 

B. CONCLUSION 

By taking a very techno-centric approach to improve efficiency and ease of work 
over the last century, society is experiencing the fruits of its labor. Machines are meant to 
help humans and improve quality of life. Unfortunately, focusing so much on technology. 
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or the human in the equation, appears to hold less importance. Some may argue the genie 
is out of the bottle and there is no putting it back.^^o Technology is on a path that cannot 
be stopped, I disagree. It is possible to rehabilitate humanity but not alone. Partnering 
with technology, and learning new innovative ways to help humans adapt quicker, can 
help decrease the hybrid phase, which will ultimately help society reach Brown’s Point 
and see the intention of advancing technology realized. 

The idea of Brown’s Point is applicable for any new advancing technology meant 
to interface with humans. Ensuring the variables and relationships between each is 
understood will help to design safe and helpful technology. It is possible with the speed 
of technology development that a reliable self-driving vehicle may come to fruition in the 
near future. Ensuring humans are as prepared as possible for this transition will allow 
self-driving vehicles to reach Brown’s Point, which is fantastic news for the safety of this 
nation’s roads and promises to save thousands of lives each year. 


Bostrom, Superintelligence: Paths, Dangers. 
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